Image processing device and image processing method

ABSTRACT

There is provided an image processing device including a decoding section that decodes an encoded stream and generates quantized transform coefficient data, and an inverse quantization section that, taking transform coefficient data as transform units to be used during inverse orthogonal transform, inversely quantizes the quantized transform coefficient data decoded by the decoding section, such that in a case where a non-square transform unit is selected, the inverse quantization section uses a non-square quantization matrix, corresponding to a non-square transform unit, that is generated from a square quantization matrix corresponding to a square transform unit.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation application of U.S. patentapplication Ser. No. 15/406,911, filed Jan. 16, 2017, which is acontinuation application of U.S. patent application Ser. No. 14/113,759,filed Oct. 24, 2013, now U.S. Pat. No. 9,641,840, which is a nationalstage entry of PCT/JP2012/059779, filed Apr. 10, 2012, which claimspriority from prior Japanese Priority Patent Application JP 2011-235828filed in the Japan Patent Office on Oct. 27, 2011, which claims priorityfrom prior Japanese Priority Patent Application JP 2011-113232 filed inthe Japan Patent Office on May 20, 2011, the entire contents of whichare hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates to an image processing device and animage processing method.

BACKGROUND ART

In H.264/AVC, one of the standard specifications for image encodingschemes, it is possible to use different quantization steps for eachcomponent of the orthogonal transform coefficients when quantizing imagedata in the High Profile or higher profile. A quantization step for eachcomponent of the orthogonal transform coefficients may be set on thebasis of a quantization matrix (also called a scaling list) defined atthe same size as the units of orthogonal transform, and a standard stepvalue.

FIG. 28 illustrates four classes of default quantization matrices whichare predefined in H.264/AVC. The matrix SL1 is the default 4×4quantization matrix for intra prediction mode. The matrix SL2 is thedefault 4×4 quantization matrix for inter prediction mode. The matrixSL3 is the default 8×8 quantization matrix for intra prediction mode.The matrix SL4 is the default 8×8 quantization matrix for interprediction mode. The user may also define one's own quantization matrixthat differs from the default matrices illustrated in FIG. 28 in thesequence parameter set or the picture parameter set. Note that in thecase where no quantization matrix is specified, a flat quantizationmatrix having an equal quantization step for all components may be used.

In High Efficiency Video Coding (HEVC), whose standardization is beingadvanced as a next-generation image encoding scheme to succeedH.264/AVC, there is introduced the concept of a coding unit (CU), whichcorresponds to a macroblock of the past (see Non-Patent Literature 1below). Furthermore, one coding unit may be split into one or moreprediction units (PUs), which signify the units of a prediction process.Intra prediction or inter prediction is then conducted on eachprediction unit. In addition, one coding unit may be split into one ormore transform units (TUs), which signify the units of orthogonaltransform. Each transform unit is then subjected to an orthogonaltransform from image data into transform coefficient data, and thetransform coefficient data is quantized. Non-Patent Literature 2 belowproposes a technique called the short distance intra prediction method,which enables the selection of comparatively small-size, non-squareprediction units (line or rectangle prediction units, for example) inintra prediction modes. In this case, the shapes of the transform unitsmay also become non-square to match the shapes of the prediction units.

CITATION LIST Non-Patent Literature

Non-Patent Literature 1: JCTVC-B205, “Test Model under Consideration”,ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 2nd Meeting: Geneva, 21-28Jul. 2010

Non-Patent Literature 2: JCTVC-E278, “CE6.b1 Report on Short DistanceIntra Prediction Method”, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 5thMeeting: Geneva, 16-23 Mar. 2011

SUMMARY OF INVENTION Technical Problem

However, as the selectable types of shapes and sizes for transform unitsincrease, the number of corresponding quantization matrices alsoincreases, and the increase in bit rate due to the quantization matricesmay instead lead to lowered coding efficiency. Consequently, it isdesirable to provide a mechanism that does not significantly lowercoding efficiency, even if there is an increase in candidates for thequantization matrix to use.

Solution to Problem

According to an embodiment of the present disclosure, there is providedan image processing device including a decoding section that decodes anencoded stream and generates quantized transform coefficient data, andan inverse quantization section that, taking transform coefficient dataas transform units to be used during inverse orthogonal transform,inversely quantizes the quantized transform coefficient data decoded bythe decoding section, such that in a case where a non-square transformunit is selected, the inverse quantization section uses a non-squarequantization matrix, corresponding to a non-square transform unit, thatis generated from a square quantization matrix corresponding to a squaretransform unit.

The above image processing device typically may be realized as an imagedecoding device that decodes images.

Further, according to an embodiment of the present disclosure, there isprovided an image processing method including decoding an encoded streamand generating quantized transform coefficient data, and takingtransform coefficient data as transform units to be used during inverseorthogonal transform, inversely quantizing the decoded quantizedtransform coefficient data, such that in a case where a non-squaretransform unit is selected, a non-square quantization matrixcorresponding to a non-square transform unit is used, the non-squarequantization matrix being generated from a square quantization matrixcorresponding to a square transform unit.

Advantageous Effects of Invention

As described above, according to the present disclosure, there isprovided a mechanism that does not significantly lower codingefficiency, even if there is an increase in candidates for thequantization matrix to use as a result of an increase in the selectabletypes of transform units.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary configuration of animage encoding device according to an embodiment.

FIG. 2 is a block diagram illustrating an example of a detailedconfiguration of the syntax processing section illustrated in FIG. 1.

FIG. 3 is a block diagram illustrating an example of a detailedconfiguration of the orthogonal transform section illustrated in FIG. 1.

FIG. 4 is a block diagram illustrating an example of a detailedconfiguration of the quantization section illustrated in FIG. 1.

FIG. 5 is an explanatory diagram illustrating an example of parametersfor generating a quantization matrix.

FIG. 6A is a first explanatory diagram for explaining the generation ofa non-square quantization matrix in a copy mode.

FIG. 6B is a second explanatory diagram for explaining the generation ofa non-square quantization matrix in a copy mode.

FIG. 6C is a third explanatory diagram for explaining the generation ofa non-square quantization matrix in a copy mode.

FIG. 7 is an explanatory diagram for explaining the generation of aquantization matrix in a transpose mode.

FIG. 8A is a first explanatory diagram for explaining a simplifiedtechnique for generating non-square quantization matrices.

FIG. 8B is a second explanatory diagram for explaining a simplifiedtechnique for generating non-square quantization matrices.

FIG. 9 is an explanatory diagram for explaining a technique forgenerating non-square quantization matrices in the case of using adefault square quantization matrix.

FIG. 10 is an explanatory diagram for explaining examples of scanpatterns for a non-square quantization matrix.

FIG. 11 is a flowchart illustrating an example of the flow of a processduring encoding according to an embodiment.

FIG. 12 is a block diagram illustrating an exemplary configuration of animage decoding device according to an embodiment.

FIG. 13 is a block diagram illustrating an example of a detailedconfiguration of the syntax processing section illustrated in FIG. 12.

FIG. 14 is a block diagram illustrating an example of a detailedconfiguration of the inverse quantization section illustrated in FIG.12.

FIG. 15 is a block diagram illustrating an example of a detailedconfiguration of the inverse orthogonal transform section illustrated inFIG. 12.

FIG. 16 is a flowchart illustrating an example of the flow of a processduring decoding according to an embodiment.

FIG. 17 is a flowchart illustrating an example of the flow of thequantization matrix generation process illustrated in FIG. 16.

FIG. 18 is an explanatory diagram for explaining a multiview codec.

FIG. 19 is an explanatory diagram for explaining an application of animage encoding process according to an embodiment to a multiview codec.

FIG. 20 is an explanatory diagram for explaining an application of animage decoding process according to an embodiment to a multiview codec.

FIG. 21 is an explanatory diagram for explaining a scalable codec.

FIG. 22 is an explanatory diagram for explaining an application of animage encoding process according to an embodiment to a scalable codec.

FIG. 23 is an explanatory diagram for explaining an application of animage encoding process according to an embodiment to a scalable codec.

FIG. 24 is a block diagram illustrating an example of a schematicconfiguration of a television.

FIG. 25 is a block diagram illustrating an example of a schematicconfiguration of a mobile phone.

FIG. 26 is a block diagram illustrating an example of a schematicconfiguration of a recording and playback device.

FIG. 27 is a block diagram illustrating an example of a schematicconfiguration of an imaging device.

FIG. 28 is an explanatory diagram illustrating default quantizationmatrices which are predefined in H.264/AVC.

DESCRIPTION OF EMBODIMENTS

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the drawings, elements that have substantiallythe same function and structure are denoted with the same referencesigns, and repeated explanation is omitted.

Also, the description will proceed in the following order.

-   1. Exemplary configuration of image encoding device according to    embodiment

1-1. Exemplary overall configuration

1-2. Exemplary configuration of syntax processing section

1-3. Exemplary configuration of orthogonal transform section

1-4. Exemplary configuration of quantization section

1-5. Exemplary parameter structure

1-6. Examples of generating non-square quantization matrices

1-7. Examples of scan patterns

-   2. Process flow during encoding according to embodiment-   3. Exemplary configuration of image decoding device according to    embodiment

3-1. Exemplary overall configuration

3-2. Exemplary configuration of syntax processing section

3-3. Exemplary configuration of inverse quantization section

3-4. Exemplary configuration of inverse orthogonal transform section

-   4. Process flow during decoding according to embodiment-   5. Application to various codecs

5-1. Multiview codec

5-2. Scalable codec

-   6. Applications-   7. Conclusion

1. Exemplary Configuration of Image Encoding Device According toEmbodiment

This section describes an exemplary configuration of an image encodingdevice according to an embodiment.

[1-1. Exemplary Overall Configuration]

FIG. 1 is a block diagram illustrating an exemplary configuration of animage encoding device 10 according to an embodiment. Referring to FIG.1, the image encoding device 10 is equipped with an analog-to-digital(A/D) conversion section 11, a reordering buffer 12, a syntax processingsection 13, a subtraction section 14, an orthogonal transform section15, a quantization section 16, a lossless encoding section 17, anaccumulation buffer 18, a rate control section 19, an inversequantization section 21, an inverse orthogonal transform section 22, anaddition section 23, a deblocking filter 24, frame memory 25, a selector26, an intra prediction section 30, a motion estimation section 40, anda mode selecting section 50.

The A/D conversion section 11 converts an image signal input in ananalog format into image data in a digital format, and outputs asequence of digital image data to the reordering buffer 12.

The reordering buffer 12 reorders the images included in the sequence ofimage data input from the A/D conversion section 11. After reorderingthe images according to a group of pictures (GOP) structure inaccordance with the encoding process, the reordering buffer 12 outputsthe reordered image data to the syntax processing section 13.

The image data output from the reordering buffer 12 to the syntaxprocessing section 13 is mapped to a bitstream in units called NetworkAbstraction Layer (NAL) units. The stream of image data includes one ormore sequences. The leading picture in a sequence is called theinstantaneous decoding refresh (IDR) picture. Each sequence includes oneor more pictures, and each picture further includes one or more slices.In H.264/AVC and HEVC, these slices are the basic units of imageencoding and decoding. The data for each slice is recognized as a VideoCoding Layer (VCL) NAL unit.

The syntax processing section 13 sequentially recognizes the NAL unitsin the stream of image data input from the reordering buffer 12, andinserts non-VCL NAL units storing header information into the stream.The non-VCL NAL units that the syntax processing section 13 inserts intothe stream include sequence parameter sets (SPSs) and picture parametersets (PPSs). The header information stored in SPS and PSP include, forexample, a parameter related to a quantization matrix later described(hereinafter referred to as a quantization matrix parameter). Note thatanother new parameter set different from SPS and PPS may be set. Forexample, the syntax processing section 13 may insert into the stream aquantization matrix parameter set (QMPS), which stores only thequantization matrix parameters. The syntax processing section 13 alsoadds a slice header (SH) at the beginning of the slices. The syntaxprocessing section 13 then outputs the stream of image data includingVCL NAL units and non-VCL NAL units to the subtraction section 14, theintra prediction section 30, and the motion estimation section 40. Adetailed configuration of the syntax processing section 13 will befurther described later.

The subtraction section 14 is supplied with the image data input fromthe syntax processing section 13, and predicted image data selected bythe mode selecting section 50 described later. The subtraction section14 calculates prediction error data, which is the difference between theimage data input from the syntax processing section 13 and the predictedimage data input from the mode selecting section 50, and outputs thecalculated prediction error data to the orthogonal transform section 15.

The orthogonal transform section 15 sets transform units in an image tobe encoded, and generates transform coefficient data by orthogonallytransforming image data for each transform unit. In the presentembodiment, the shape of a transform unit set by the orthogonaltransform section 15 may be square or non-square. The edge size of asquare transform unit may be a size such as 4 pixels, 8 pixels, 16pixels, or 32 pixels. The size of the long edge of a non-squaretransform unit likewise may be a size such as 4 pixels, 8 pixels, 16pixels, or 32 pixels, and the ratio of the size of the long edge to thesize of the short edge may be a ratio such as 2:1, 4:1, or 8:1. Theimage data subjected to an orthogonal transform by the orthogonaltransform section 15 is prediction error data input from the subtractionsection 14. The orthogonal transform by the orthogonal transform section15 may be conducted according to an arbitrary orthogonal transformscheme, such as a discrete cosine transform (DCT) scheme, a discretesine transform (DST) scheme, a Hadamard transform scheme, or aKarhunen-Loeve transform scheme, for example. The orthogonal transformsection 15 outputs transform coefficient data transformed fromprediction error data via an orthogonal transform process to thequantization section 16. A detailed configuration of the orthogonaltransform section 15 will be further described later.

The quantization section 16 uses a quantization matrix corresponding toeach transform unit to quantize the transform coefficient data in eachtransform unit input from the orthogonal transform section 15, andoutputs the quantized transform coefficient data (hereinafter referredto as quantized data) to the lossless encoding section 17 and theinverse quantization section 21. The bit rate of the quantized data iscontrolled on the basis of a rate control signal from the rate controlsection 19. The quantization matrix used by the quantization section 16is defined in an SPS, PPS, or some other parameter set, and may bespecified in the slice header for each slice. It is also possible to usea default quantization matrix like those exemplified in FIG. 28 insteadof defining a quantization matrix in a parameter set. In this case, theparameters for defining a quantization matrix become unnecessary. Adetailed configuration of the quantization section 16 will be furtherdescribed later.

The lossless encoding section 17 generates an encoded stream by encodingquantized data input from the quantization section 16. In addition, thelossless encoding section 17 encodes quantization matrix parametersinserted into a stream by the syntax processing section 13, andmultiplexes the encoded parameters into the encoded stream. Furthermore,the lossless encoding section 17 encodes information about intraprediction or information about inter prediction input from the modeselecting section 50, and multiplexes the encoded information into theencoded stream. Typically, the encoding by the lossless encoding section17 is lossless variable-length encoding based on a scheme such asarithmetic coding, Golomb coding, or Huffman coding. The losslessencoding section 17 then outputs the encoded stream thus generated tothe accumulation buffer 18.

The accumulation buffer 18 uses a storage medium such as semiconductormemory to temporarily buffer the encoded stream input from the losslessencoding section 17. The accumulation buffer 18 then outputs the encodedstream thus buffered to a transmission section not illustrated (such asa communication interface or a connection interface with peripheralequipment, for example), at a rate according to the bandwidth of thetransmission channel.

The rate control section 19 monitors the free space in the accumulationbuffer 18. Then, the rate control section 19 generates a rate controlsignal according to the free space in the accumulation buffer 18, andoutputs the generated rate control signal to the quantization section16. For example, when there is not much free space in the accumulationbuffer 18, the rate control section 19 generates a rate control signalfor lowering the bit rate of the quantized data. Also, when there issufficient free space in the accumulation buffer 18, for example, therate control section 19 generates a rate control signal for raising thebit rate of the quantized data.

The inverse quantization section 21 performs an inverse quantizationprocess on the quantized data input from the quantization section 16,using the same quantization matrix as the one set during thequantization process by the quantization section 16. The inversequantization section 21 then outputs transform coefficient data acquiredby the inverse quantization process to the inverse orthogonal transformsection 22.

The inverse orthogonal transform unit 22 restores the prediction errordata by applying an inverse orthogonal transform to the transformcoefficient data input from the inverse quantization section 21. Theorthogonal transform method used by the inverse orthogonal transformsection 22 is equal to the method selected during the orthogonaltransform process by the orthogonal transform section 15. The inverseorthogonal transform section 22 then outputs the restored predictionerror data to the addition section 23.

The addition section 23 adds the restored prediction error data inputfrom the inverse orthogonal transform section 22 and the predicted imagedata input from the mode selecting section 50 to thereby generatedecoded image data. Then, the addition section 23 outputs the decodedimage data thus generated to the deblocking filter 24 and the framememory 25.

The deblocking filter 24 applies filtering to reduce blocking artifactsproduced at the time of image encoding. The deblocking filter 24 removesblocking artifacts by filtering the decoded image data input from theaddition section 23, and outputs the decoded image data thus filtered tothe frame memory 25.

The frame memory 25 uses a storage medium to store the decoded imagedata input from the addition section 23 and the decoded image data afterfiltering input from the deblocking filter 24.

The selector 26 reads, from the frame memory 25, unfiltered decodedimage data to be used for intra prediction, and supplies the decodedimage data thus read to the intra prediction section 30 as referenceimage data. Also, the selector 26 reads, from the frame memory 25, thefiltered decoded image data to be used for inter prediction, andsupplies the decoded image data thus read to the motion estimationsection 40 as reference image data.

The intra prediction section 30 performs an intra prediction process ineach intra prediction mode, on the basis of the image data to be encodedthat is input from the syntax processing section 13, and the decodedimage data supplied via the selector 26. For example, the intraprediction section 30 evaluates the prediction result of each intraprediction mode using a predetermined cost function. Then, the intraprediction section 30 selects the intra prediction mode yielding thesmallest cost function value, that is, the intra prediction modeyielding the highest compression ratio, as the optimal intra predictionmode. The intra prediction section 30 then outputs the predicted imagedata, information about intra prediction including the selected optimalintra prediction mode or the like, and the cost function value, to themode selecting section 50.

The motion estimation section 40 performs an inter prediction process(prediction process between frames) on the basis of image data to beencoded that is input from the syntax processing section 13, and decodedimage data supplied via the selector 26. For example, the motionestimation section 40 evaluates the prediction result of each predictionmode using a predetermined cost function. Then, the motion estimationsection 40 selects the prediction mode yielding the smallest costfunction value, that is, the prediction mode yielding the highestcompression ratio, as the optimal prediction mode. The motion estimationsection 40 generates predicted image data according to the optimalprediction mode. The motion estimation section 40 outputs the predictedimage data, information about inter prediction including the selectedoptimal prediction mode or the like, and the cost function value, to themode selecting section 50.

The mode selecting section 50 compares the cost function value relatedto intra prediction input from the intra prediction section 30 to thecost function value related to inter prediction input from the motionestimation section 40. Then, the mode selecting section 50 selects theprediction method with the smaller cost function value between intraprediction and inter prediction. In the case of selecting intraprediction, the mode selecting section 50 outputs the information aboutintra prediction to the lossless encoding section 17, and also outputsthe predicted image data to the subtraction section 14 and the additionsection 23. Also, in the case of selecting inter prediction, the modeselecting section 50 outputs the information about inter predictiondescribed above to the lossless encoding section 17, and also outputsthe predicted image data to the subtraction section 14 and the additionsection 23.

[1-2. Exemplary Configuration of Syntax Processing Section]

FIG. 2 is a block diagram illustrating an example of a detailedconfiguration of the syntax processing section 13 of the image encodingdevice 10 illustrated in FIG. 1. Referring to FIG. 2, the syntaxprocessing section 13 includes a settings storage section 132, aparameter generating section 134, and an inserting section 136.

(1) Settings Storage Section

The settings storage section 132 stores various settings used for theencoding process by the image encoding device 10. For example, thesettings storage section 132 stores information such as a profile foreach sequence in the image data, the encoding mode for each picture,data regarding the GOP structure, as well as coding unit, predictionunit, and transform unit settings. Also, in the present embodiment, thesettings storage section 132 stores settings regarding quantizationmatrices used by the quantization section 16 (and the inversequantization section 21). These settings may be predetermined for eachslice, typically on the basis of offline image analysis.

(2) Parameter Generating Section

The parameter generating section 134 generates parameters definingsettings stored by the settings storage section 132, and outputs thegenerated parameters to the inserting section 136.

For example, in the present embodiment, the parameter generating section134 generates quantization matrix parameters for generating quantizationmatrices used as candidates by the quantization section 16. Thequantization matrix candidates used by the quantization section 16include quantization matrices corresponding to each of the types oftransform units which may be set in an image. In the present embodiment,the types of transform units are classified at least according tocombinations of transform unit shapes and sizes. An example ofquantization matrix parameters generated by the parameter generatingsection 134 will be further described later.

(3) Inserting Section

The inserting section 136 inserts header information, such as SPSs,PPSs, and slice headers that respectively include parameter groupsgenerated by the parameter generating section 134, into the stream ofimage data input from the reordering buffer 12. The header informationinserted into the stream of image data by the inserting section 136includes the quantization matrix parameters generated by the parametergenerating section 134. The inserting section 136 then outputs thestream of image data with inserted header information to the subtractionsection 14, the intra prediction section 30, and the motion estimationsection 40.

[1-3. Exemplary Configuration of Orthogonal Transform Section]

FIG. 3 is a block diagram illustrating an example of a detailedconfiguration of the orthogonal transform section 15 of the imageencoding device 10 illustrated in FIG. 1. Referring to FIG. 3, theorthogonal transform section 15 includes a transform unit settingsection 152 and an orthogonal transform computing section 154.

(1) Transform Unit Setting Section

The transform unit setting section 152 sets a transform unit in animage, to be used when orthogonally transforming image data to beencoded. The shape of a transform unit set by the transform unit settingsection 152 may be square or non-square. For example, in the case wherethe intra prediction section 30 uses the short distance intra predictionmethod discussed earlier, the transform unit setting section 152 may seta non-square transform unit of the same size as a prediction unit in animage when a non-square prediction unit is selected as the predictionunit.

(2) Orthogonal Transform Computing Section

The orthogonal transform computing section 154 generates transformcoefficient data by orthogonally transforming prediction error datainput from the subtraction section 14 for each transform unit set in animage by the transform unit setting section 152. The orthogonaltransform computing section 154 then outputs the generated transformcoefficient data to the quantization section 16. In addition, thetransform unit setting section 152 outputs transform unit informationspecifying the set transform unit to the quantization section 16.

[1-4. Exemplary Configuration of Quantization Section]

FIG. 4 is a block diagram illustrating an example of a detailedconfiguration of the quantization section 16 of the image encodingdevice 10 illustrated in FIG. 1. Referring to FIG. 4, the quantizationsection 16 includes a quantization matrix setting section 162 and aquantization computing section 164.

(1) Quantization Matrix Setting Section

For each transform unit set by the orthogonal transform section 15, thequantization matrix setting section 162 sets a quantization matrix forquantizing transform coefficient data generated by orthogonal transform.For example, the quantization matrix setting section 162 first acquirestransform unit information from the orthogonal transform section 15. Thetransform unit information may be information specifying the position ofa partition dividing each coding unit into one or more transform units.Additionally, the transform unit information may be informationspecifying a prediction unit instead of a transform unit in the casewhere the prediction units and the transform units are equal.

The quantization matrix setting section 162 recognizes the shape andsize of each transform unit from the acquired transform unitinformation, and sets a quantization matrix corresponding to therecognized shape and size for each transform unit. As an example, an8-row, 8-column quantization matrix is set for an 8×8 pixel transformunit, a 2-row, 8-column quantization matrix is set for a 2×8 pixeltransform unit, and an 8-row, 2-column quantization matrix is set for an8×2 pixel transform unit. The quantization matrix setting section 162may also set for each transform unit a quantization matrix that differsfor every combination of prediction mode (intra prediction/interprediction) and signal component (Y/Cb/Cr), for example. Thequantization step of the set quantization matrix may also be adjustedaccording to a rate control signal from the rate control section 19.

(2) Orthogonal Transform Computing Section

The orthogonal transform computing section 154 uses the orthogonaltransform method selected by the transform method selecting section 152to transform prediction error data input from the subtraction section 14into transform coefficient data for each transform unit. The orthogonaltransform computing section 154 then outputs the transformed transformcoefficient data to the quantization section 16. The transform methodselecting section 152 also outputs transform method informationexpressing the orthogonal transform method selected for each transformunit to the quantization section 16.

[1-5. Exemplary Parameter Structure]

FIG. 5 illustrates an example of parameters related to non-squarequantization matrices from among the quantization matrix parametersgenerated by the parameter generating section 134 of the syntaxprocessing section 13. Note that the parameters related to squarequantization matrices may be parameters similar to those of an existingimage coding scheme such as H.264/AVC.

Referring to FIG. 5, the quantization matrix parameters include a“non-square matrix flag” and parameter groups for generating eachnon-square quantization matrix.

The “non-square matrix flag” is a flag expressing whether or not to usenon-square quantization matrices. In the case where the non-squarematrix flag indicates “0: No”, non-square quantization matrices are notused, and thus the other parameters illustrated in FIG. 5 for generatingnon-square quantization matrices are omitted. On the other hand, in thecase where the non-square matrix flag indicates “1: Yes”, non-squarequantization matrices may be used, and these quantization matrices willbe generated on the decoding side on the basis of the parametersdescribed below.

The “generation mode” is one parameter for generating a non-squarequantization matrix. The “generation mode” is a classificationexpressing how to generate a non-square quantization matrix. As anexample, the generation mode classification may take one of thefollowing values:

-   0: Full scan mode-   1: Copy mode-   2: Transpose mode

(1) Full Scan Mode

If the DST generation mode is “0: Full scan mode”, the quantizationmatrix parameters additionally include “differential data”. The“differential data” is data defining a non-square quantization matrix.The “differential data” may be data obtained by converting all elementsof a non-square quantization matrix into a linear array according to agiven scan pattern, and encoding that linear array in differentialpulse-code modulation (DPCM) format, for example.

(2) Copy Mode

If the generation mode is “1: Copy mode”, a non-square quantizationmatrix is generated by copying some rows or columns of a correspondingsquare quantization matrix into the respective rows or columns of anon-square quantization matrix. The square quantization matrixcorresponding to a particular non-square quantization matrix may be asquare quantization matrix having an edge size that is equal to the sizeof the long edge of that non-square quantization matrix. In this case,the quantization matrix parameters additionally include a “designationmode”. The “designation mode” is a classification expressing how tospecify the rows or columns of a square quantization matrix that will beused as the copy source rows or columns. As an example, theclassification may take one of the following values:

-   0: Default-   1: Copy source ID-   2: Direction+copy source ID

If the designation mode is “0: Default”, rows or columns at predefined,default positions are treated as the copy source rows or columns. Forexample, if the size of the non-square quantization matrix beingprocessed is 2 rows by 8 columns (2×8), the size of the correspondingsquare quantization matrix is 8 rows by 8 columns (8×8). In this case,the 0th and 4th rows, or the 0th and 1st rows, of the 8×8 quantizationmatrix may be the default copy source. As another example, if the sizeof the non-square quantization matrix being processed is 8×2, the sizeof the corresponding square quantization matrix is likewise 8×8. In thiscase, the 0th and 4th columns, or the 0th and 1st columns, of the 8×8quantization matrix may be the default copy source. Note that theuppermost row and the leftmost column of a matrix are herein taken to bethe 0th row and the 0th column, respectively.

If the designation mode is “1: Copy source ID”, the quantization matrixparameters additionally include a “copy source ID”. The “copy source ID”expresses one or more row IDs or column IDs for specifying the positionsof the rows or columns of a corresponding square quantization matrixthat will be used as the copy source rows or columns. For example,assume that the size of the non-square quantization matrix beingprocessed is 2×8, and that the “copy source ID” indicates “0” and “3”.In this case, the 0th and 3rd rows of the 8×8 square quantization matrixbecome the copy source. Note that with this designation scheme, if thelong edge of the non-square quantization matrix being processed is ahorizontal edge, some rows of a corresponding square quantization matrixbecome the copy source. Meanwhile, if the long edge of the non-squarequantization matrix being processed is a vertical edge, some columns ofa corresponding square quantization matrix become the copy source.

If the designation mode is “2: Direction+copy source ID”, it is possibleto designate rows of a corresponding square quantization matrix as thecopy source, and also possible to designate columns as the copy source,irrespective of the direction of the long edge of the non-squarequantization matrix being processed. In this case, the quantizationmatrix parameters additionally include a “copy source ID” and a “copysource direction”. The “copy source direction” is a classification forspecifying whether to take the rows of a corresponding squarequantization matrix as the copy source, or take the columns as the copysource. As an example, the copy source direction classification may takeone of the following values:

-   0: Same direction-   1: Different direction

For example, assume that the size of the non-square quantization matrixbeing processed is 2×8, the “copy source ID” indicates “0” and “3”, andthe “copy source direction” indicates “0: Same direction”. In this case,the 0th and 3rd rows of the 8×8 square quantization matrix become thecopy source. On the other hand, in the case where the “copy sourcedirection” indicates “1: Different direction” under similar conditions,the 0th and 3rd columns of the 8×8 square quantization matrixrespectively become the copy source for the 0th and 1st rows of the 2×8non-square quantization matrix.

(3) Transpose Mode

If the generation mode is “2: Transpose mode”, the non-squarequantization matrix being processed is calculated as the transpose ofanother non-square quantization matrix whose long edge size and shortedge size are inversed. For example, an 8×2 quantization matrix may becalculated as the transpose of a 2×8 quantization matrix.

“Residual data” may be included in the quantization matrix parameters inthe case where the generation mode is “1: Copy mode” or “2: Transposemode”. The residual data may be data obtained by converting the residualfor all elements of a quantization matrix generated by copying ortransposing versus the quantization matrix actually being used into alinear array using a given scan pattern.

As discussed earlier, the quantization matrix parameters exemplified inFIG. 5 may be inserted into the SPS or PPS, or a parameter set differentfrom these parameter sets. Note that these quantization matrixparameters are merely one example. In other words, some of theparameters among the above quantization matrix parameters may also beomitted, while other parameters may also be added. In addition, theparameters other than the “non-square matrix flag” and the “residualdata” exemplified in FIG. 5 may be respectively defined for each type ofnon-square quantization matrix, or commonly defined over multiple typesof non-square quantization matrices.

For example, the question of whether to generate non-square quantizationmatrices in copy mode, and also which rows or columns to copy, may bepredefined as a specification shared between the encoder and thedecoder. In this case, it is possible to not insert parameters such asthe “generation mode”, the “designation mode”, the “copy source ID”, andthe “copy source direction” into the parameter set, thereby decreasingoverhead and potentially improving coding efficiency.

Additionally, the quantization matrix parameters exemplified in FIG. 5other than the “non-square matrix flag” may be omitted in the case ofusing a default quantization matrix as the corresponding squarequantization matrix, even when the “non-square matrix flag” indicates“1: Yes”. In such cases, a predefined, default quantization matrix mayalso be used for the non-square quantization matrix.

[1-6. Examples of Generating Non-Square Quantization Matrices]

FIGS. 6A, 6B and 6C illustrate examples of generating non-squarequantization matrices in copy mode.

The right side of FIG. 6A illustrates a 2×8 non-square quantizationmatrix M_(NS28), which is the quantization matrix being generated.Meanwhile, the left side of FIG. 6A illustrates an 8×8 squarequantization matrix M_(S8) that corresponds to the quantization matrixM_(NS28). Also, “0: Default” is specified as the designation mode. Thepredefined, default copy source is assumed to be the 0th and 4th rows.In this case, the quantization matrix M_(NS28) may be generated bycopying the 0th row of the quantization matrix M_(S8) into the 0th row,and the 4th row of the quantization matrix M_(S8) into the 1st row. Inaddition, residual may be added as necessary to each element of thecopied quantization matrix M_(NS28).

In this way, by enabling a non-square quantization matrix to begenerated by copying rows or columns from a square quantization matrix,it is possible to moderate increases in bit rate caused by using anon-square quantization matrix. In addition, taking the positions of thecopy source rows or columns to be default positions avoids increases inbit rate due to specifying the positions of the copy source rows orcolumns.

The right side of FIG. 6B illustrates an 8×2 non-square quantizationmatrix M_(NS82), which is the quantization matrix being generated.Meanwhile, the left side of FIG. 6B illustrates an 8×8 squarequantization matrix M_(S8) that corresponds to the quantization matrixM_(NS82). Also, “1: Copy source ID” is specified as the designationmode, and (0, 6) is specified as the copy source ID. In this case, thequantization matrix M_(NS82) may be generated by copying the 0th row ofthe quantization matrix M_(S8) into the 0th row, and the 6th row of thequantization matrix M_(S8) into the 1st row. In addition, residual maybe added as necessary to each element of the copied quantization matrixM_(NS82).

In this way, by specifying the positions of copy source rows or columnswhen generating a non-square quantization matrix from a squarequantization matrix, it becomes possible to further reduce the residualof the copied matrix versus the matrix actually being used, and moderatethe bit rate for the residual data. Thus, it is possible to moreeffectively moderate increases in bit rate caused by using a non-squarequantization matrix.

The right side of FIG. 6C illustrates a 2×4 non-square quantizationmatrix M_(NS24), which is the quantization matrix being generated.Meanwhile, the left side of FIG. 6C illustrates a 4×4 squarequantization matrix M_(S4) that corresponds to the quantization matrixM_(NS24). Also, “2: Copy source direction+copy source ID” is specifiedas the designation mode, (0, 3) is specified as the copy source ID, and“1: Different direction” is specified as the copy source direction. Inthis case, the quantization matrix M_(NS24) may be generated by copyingthe 0th column of the quantization matrix M_(S4) into the 0th row, andthe 3rd column of the quantization matrix M_(S4) into the 1st row. Inaddition, residual may be added as necessary to each element of thecopied quantization matrix M_(NS24).

In this way, by enabling the specification of both the rows and columnsof a square quantization matrix as the copy source, irrespective of theshape of the non-square quantization matrix, it is possible to broadenthe selection range of a copy source, particularly in the case of usinga quantization matrix having asymmetric element values. Thus, it ispossible to minimize the residual of the copied matrix versus the matrixactually being used.

FIG. 7 illustrates an example of generating a non-square quantizationmatrix in transpose mode.

The right side of FIG. 7 illustrates a 2×8 non-square quantizationmatrix M_(NS28), which is the quantization matrix being generated.Meanwhile, the left side of FIG. 7 illustrates an 8×2 non-squarequantization matrix M_(NS82). In this case, the quantization matrixM_(NS28) may be generated as the transpose of the quantization matrixM_(NS82) (M_(NS82) ^(T)). In addition, residual may be added asnecessary to each element of the transposed quantization matrixM_(NS28).

In this way, in transpose mode, it is possible to generate a non-squarequantization matrix similar to the case of using copy mode from another,symmetric non-square quantization matrix, without requiring parameterssuch as the designation mode, copy source ID, and copy source direction.Consequently, it is possible to further reduce the bit rate for anon-square quantization matrix.

FIGS. 8A and 8B are explanatory diagram for explaining a simplifiedtechnique for generating non-square quantization matrices.

The right side of FIG. 8A illustrates two non-square quantizationmatrices M_(NS416) and M_(NS164), which are the quantization matricesbeing generated. Meanwhile, the left side of FIG. 8A illustrates acorresponding 8×8 square quantization matrix M_(S16). Herein, thequestion of whether to generate non-square quantization matrices in copymode, and also which rows/columns to copy, is assumed to be known byboth the encoder and the decoder (for example, predefined as aspecification). Consequently, the “generation mode”, “designation mode”,“copy source ID”, and “copy source direction” are not encoded asquantization parameters. The positions of the predefined, default copysource rows/columns may be any positions. In the example in FIG. 8A, thepositions of the predefined, default copy source rows and columns areassumed to be the positions of every fourth row starting from the 0throw and every fourth column starting from the 0th column, respectively(the shaded portions of the drawing). In this case, the quantizationmatrix M_(NS416) may be generated by copying the 0th, 4th, 8th, and 12throws of the quantization matrix M_(S16) into the four rows. Thequantization matrix M_(NS164) may be generated by copying the 0th, 4th,8th, and 12th columns of the quantization matrix M_(S16) into the fourcolumns.

The right side of FIG. 8B illustrates two non-square quantizationmatrices M_(NS832) and M_(NS328), which are the quantization matricesbeing generated. Meanwhile, the left side of FIG. 8B illustrates acorresponding 32×32 square quantization matrix M_(S32). Similarly to theexample in FIG. 8A, the “generation mode”, “designation mode”, “copysource ID”, and “copy source direction” are not encoded as quantizationparameters. Similarly to the example in FIG. 8A, the positions of thepredefined, default copy source rows and columns are assumed to be thepositions of every fourth row starting from the 0th row and every fourthcolumn starting from the 0th column, respectively (the shaded portionsof the drawing). In this case, the quantization matrix M_(NS832) may begenerated by copying the 0th, 4th, 8th, 12th, 16th, 20th, 24th, and 28throws of the quantization matrix M_(S32) into the eight rows. Thequantization matrix M_(NS328) may be generated by copying the 0th, 4th,8th, 12th, 16th, 20th, 24th, and 28th columns of the quantization matrixM_(S32) into the eight columns.

Note that the positions of the default copy source rows/columns may bepositions assigned at equal intervals according to a ratio of the edgesize of the matrix, as in the examples in FIGS. 8A and 8B. Otherwise, Nconsecutive rows/columns from the top edge or left edge may be used, forexample. Also, instead of copying rows/columns at default positions,four elements at default positions in a square quantization matrix maybe copied to into the elements at the four vertex positions of anon-square quantization matrix, with the remaining element beinginterpolated (using a technique such as linear interpolation, forexample). In addition, N elements at default positions in a squarequantization matrix (where N is equal to the number of all elements in anon-square quantization matrix) may be respectively copied into theelements at corresponding positions in the non-square quantizationmatrix.

In this way, by predefining whether to generate a non-squarequantization matrix from a square quantization matrix in copy mode, andalso how to copy the matrices, it is possible to omit much of theencoding of quantization matrix parameters for a non-square quantizationmatrix. Thus, it is possible to reduce transmission overhead, while alsoreducing the complexity of the encoder and decoder configurations.

FIG. 9 is an explanatory diagram for explaining a technique forgenerating non-square quantization matrices in the case of using adefault square quantization matrix.

The left side of FIG. 9 illustrates an 8×8 default square quantizationmatrix MS8 def. In the case of using such a default square quantizationmatrix M_(S8def), the corresponding non-square quantization matrices mayalso be default matrices. The right side of FIG. 9 illustrates twodefault non-square quantization matrices M_(NS28def) and M_(NS82def).Instead of being generated from the square quantization matrix M_(S8def)(using a technique such as copying rows or columns), the non-squarequantization matrices M_(NS28def) and M_(NS82def) may be predefined as ashared specification, and stored in memory in both the encoder and thedecoder, for example.

In this way, in the case of using a default square quantization matrix,taking the non-square quantization matrices to also be defaultquantization matrices makes it possible to decrease the quantizationmatrix parameters to encode, and reduce transmission overhead.Additionally, it is possible to reduce the complexity of the encoder anddecoder configurations.

[1-7. Examples of Scan Patterns]

With the quantization matrix parameters exemplified in FIG. 5,differential data in full scan mode, as well as residual data in copymode and transpose mode, are generated by linearizing a two-dimensionalmatrix according to some scan pattern. The scan patterns generally usedwhen linearizing the elements of a square quantization matrix are whatare called zigzag scans. In contrast, when linearizing the elements of anon-square quantization matrix, scan patterns that differ from scanpatterns resembling zigzag scans may also be used.

FIG. 10 illustrates three examples of scan patterns which may be usedfor a non-square quantization matrix.

The first example on the left side of FIG. 10 is a scan pattern thatresembles a zigzag scan. Herein, this scan pattern is also called azigzag scan. In a zigzag scan, a high correlation is assumed betweenelements positioned on the diagonals from the upper-right to thelower-left in the quantization matrix, and a scan sequence is determinedsuch that elements in highly correlated element groups are consecutivelyscanned.

The second example in the center of FIG. 10 is a scan pattern thatconsecutively scans element groups arranged in the direction of the longedge of a non-square quantization matrix. This scan pattern is called along edge priority scan. A long edge priority scan decreases differencesbetween consecutively scanned elements and optimizes the post-DPCMlinear array in the case of high correlation between elements arrangedin the direction of the long edge in a quantization matrix.

The third example on the right side of FIG. 10 is a scan pattern thatconsecutively scans element groups arranged in the direction of theshort edge of a non-square quantization matrix. This scan pattern iscalled a short edge priority scan. A short edge priority scan decreasesdifferences between consecutively scanned elements and optimizes thepost-DPCM linear array in the case of high correlation between elementsarranged in the direction of the short edge in a quantization matrix.

The scan pattern used for a non-square quantization matrix may be anystatically defined scan pattern from among the scan patterns exemplifiedin FIG. 10. Otherwise, the scan pattern that optimizes the codingefficiency may be adaptively selected from among multiple scan patterncandidates for each sequence, each picture, or each slice. In this case,an identifier for identifying the scan pattern to use (such as the scanID (scan_id) exemplified in FIG. 10) may be included in a parameter setsuch as the SPS, PPS, or slice header.

2. Process Flow During Encoding According to Embodiment

FIG. 11 is a flowchart illustrating an example of the flow of a processduring encoding by an image encoding device 10 according to the presentembodiment. Note that for the sake of the clarity of the descriptionherein, only the processing steps relating to the orthogonal transformand quantization of image data are illustrated.

Referring to FIG. 11, first, the transform unit setting section 152 ofthe orthogonal transform section 15 recognizes the shape and size ofeach transform unit to set in image data to encode, and sets transformunits within the image (step S110).

Next, the orthogonal transform computing section 154 generates transformcoefficient data by orthogonally transforming image data (predictionerror data input from the subtraction section 14) for each transformunit set by the transform unit setting section 152 (step S120).

Next, the quantization matrix setting section 162 of the quantizationsection 16 sets a quantization matrix for each transform unit accordingto the shape and size of the set transform unit (step S130).

Next, the quantization computing section 164 uses the quantizationmatrix set by the quantization matrix setting section 162 to quantizethe transform coefficient data input from the orthogonal transformcomputing section 154 for each transform unit (step S140).

The lossless encoding section 17 then generates an encoded stream byencoding quantized data input from the quantization computing section164, and also encodes and multiplexes quantization matrix parametersinto the encoded stream (step S150).

Typically, these processing steps may be repeated for all transformunits within the image to encode.

3. Exemplary Configuration of Image Decoding Device According toEmbodiment

This section describes an exemplary configuration of an image decodingdevice according to an embodiment.

[3-1. Exemplary Overall Configuration]

FIG. 12 is a block diagram illustrating an exemplary configuration of animage decoding device 60 according to an embodiment. Referring to FIG.12, the image decoding device 60 is equipped with a syntax processingsection 61, a lossless decoding section 62, an inverse quantizationsection 63, an inverse orthogonal transform section 64, an additionsection 65, a deblocking filter 66, a reordering buffer 67, adigital-to-analog (D/A) conversion section 68, frame memory 69,selectors 70 and 71, an intra prediction section 80, and a motioncompensation section 90.

The syntax processing section 61 acquires header information such asSPSs, PPSs, and slice headers from an encoded stream input via atransmission channel, and recognizes various settings for a decodingprocess by the image decoding device 60 on the basis of the acquiredheader information. For example, in the present embodiment, the syntaxprocessing section 61 generates candidates for a quantization matrix tobe possibly used during an inverse quantization process by the inversequantization section 63 on the basis of quantization matrix parametersincluded in each parameter set. A detailed configuration of the syntaxprocessing section 61 will be further described later.

The lossless decoding section 62 decodes the encoded stream input fromthe syntax processing section 63 according to the coding method used atthe time of encoding. The lossless decoding section 62 then outputs thedecoded quantization data to the inverse quantization section 62. Inaddition, the lossless decoding section 62 outputs information aboutintra prediction included in the header information to the intraprediction section 80, and outputs information about inter prediction tothe motion compensation section 90.

The inverse quantization section 63 uses a quantization matrixcorresponding to the shape and size of each transform unit from amongthe quantization matrix candidates generated by the syntax processingsection 61 to inversely quantize the quantization data decoded by thelossless decoding section 62 (that is, quantized transform coefficientdata). A detailed configuration of the inverse quantization section 63will be further described later.

The inverse orthogonal transform section 64 generates prediction errordata by inverse orthogonally transforming inversely quantized transformcoefficient data for each transform unit set within an image to decode.The shape of a transform unit which may be set in the present embodimentis square or non-square, as discussed earlier. The inverse orthogonaltransform section 64 then outputs the generated prediction error data tothe addition section 65.

The addition section 65 adds the prediction error data input from theinverse orthogonal transform section 64 to predicted image data inputfrom the selector 71 to thereby generate decoded image data. Then, theaddition section 65 outputs the decoded image data thus generated to thedeblocking filter 66 and the frame memory 69.

The deblocking filter 66 removes blocking artifacts by filtering thedecoded image data input from the addition section 65, and outputs thedecoded image data thus filtered to the reordering buffer 67 and theframe memory 69.

The reordering buffer 67 generates a chronological sequence of imagedata by reordering images input from the deblocking filter 66. Then, thereordering buffer 67 outputs the generated image data to the D/Aconversion section 68.

The D/A conversion section 68 converts the image data in a digitalformat input from the reordering buffer 67 into an image signal in ananalog format. Then, the D/A conversion section 68 causes an image to bedisplayed by outputting the analog image signal to a display (notillustrated) connected to the image decoding device 60, for example.

The frame memory 69 uses a storage medium to store the unfiltereddecoded image data input from the addition section 65 and the filtereddecoded image data input from the deblocking filter 66.

The selector 70 switches the output destination of the image data fromthe frame memory 69 between the intra prediction section 80 and themotion compensation section 90 for each block in the image according tomode information acquired by the lossless decoding section 62. Forexample, in the case where an intra prediction mode is specified, theselector 70 outputs the unfiltered decoded image data that is suppliedfrom the frame memory 69 to the intra prediction section 80 as referenceimage data. Also, in the case where an inter prediction mode isspecified, the selector 70 outputs the filtered decoded image data thatis supplied from the frame memory 69 to the motion compensation section90 as reference image data.

The selector 71 switches the output source of predicted image data to besupplied to the addition section 65 between the intra prediction section80 and the motion compensation section 90 for each block in the imageaccording to the mode information acquired by the lossless decodingsection 62. For example, in the case where an intra prediction mode isspecified, the selector 71 supplies the addition section 65 with thepredicted image data output from the intra prediction section 80. In thecase where an inter prediction mode is specified, the selector 71supplies the addition section 65 with the predicted image data outputfrom the motion compensation section 90.

The intra prediction section 80 performs in-picture prediction of pixelvalues on the basis of the information about intra prediction input fromthe lossless decoding section 62 and the reference image data from theframe memory 69, and generates predicted image data. Then, the intraprediction section 80 outputs the predicted image data thus generated tothe selector 71.

The motion compensation section 90 performs a motion compensationprocess on the basis of the information about inter prediction inputfrom the lossless decoding section 62 and the reference image data fromthe frame memory 69, and generates predicted image data. Then, themotion compensation section 90 outputs the predicted image data thusgenerated to the selector 71.

[3-2. Exemplary Configuration of Syntax Processing Section]

FIG. 13 is a block diagram illustrating an example of a detailedconfiguration of the syntax processing section 61 of the image decodingdevice 60 illustrated in FIG. 12. Referring to FIG. 13, the syntaxprocessing section 61 includes a parameter acquisition section 212 and ageneration section 214.

(1) Parameter Acquisition Section

The parameter acquisition section 212 recognizes header information suchas SPSs, PPSs, and slice headers from the stream of image data, andacquires parameters included in the header information. For example, inthe present embodiment, the parameter acquisition section 212 acquiresquantization matrix parameters defining a quantization matrix from eachparameter set. The parameter acquisition section 212 then outputs theacquired parameters to the generation section 214. The parameteracquisition section 212 also outputs the stream of image data to thelossless decoding section 62.

(2) Generation Section

The generation section 214 generates quantization matrix candidateswhich may be used by the inverse quantization section 63, on the basisof the quantization matrix parameters acquired by the parameteracquisition section 212. In the present embodiment, the quantizationmatrices generated by the generation section 214 include quantizationmatrices that correspond to each type (that is, each combination ofshape and size) of transform unit, which is the unit of inverseorthogonal transform by the inverse orthogonal transform section 64.

More specifically, in the case where the default quantization matrix isnot used, for example, the generation section 214 generates squarequantization matrices of various sizes on the basis of a definition inthe parameter set or the header of the encoded stream. The generationsection 214 also generates non-square quantization matrices in the casewhere the parameter set or the header of the encoded stream includes aflag indicating that a non-square quantization matrix is to be used (forexample, the non-square matrix flag discussed earlier). Non-squarequantization matrices may be generated according to any of the full scanmode, copy mode, and transpose mode discussed earlier.

For example, in copy mode, the generation section 214 generates anon-square quantization matrix by copying rows or columns from acorresponding square quantization matrix. The question of which rows orcolumns to copy from the square quantization matrix may be specified bya copy source ID and copy source direction in the quantization matrixparameters. Meanwhile, in the case where the rows or columns to copy arenot specified, rows or columns at predefined, default positions may betreated as the copy source.

Additionally, in full scan mode, the generation section 214 generates anon-square quantization matrix on the basis of a definition usingdifferential data in DPCM format, instead of generating a non-squarequantization matrix from a square quantization matrix.

Additionally, in transpose mode, the generation section 214 generates anon-square quantization matrix as the transpose of another non-squarequantization matrix having a symmetric shape to the non-squarequantization matrix. In the cases of copy mode and transpose mode, thegeneration section 214 may furthermore add residual defined in thequantization matrix parameters to each element of the copiedquantization matrix or the transposed quantization matrix.

Note that in the case where the square quantization matrix correspondingto a particular non-square quantization matrix is a default quantizationmatrix, the generation section 214 uses a predefined, default non-squarequantization matrix as the non-square quantization matrix.

The generation section 214 outputs quantization matrix candidatesgenerated in this way to the inverse quantization section 63.

[3-3. Exemplary Configuration of Inverse Quantization Section]

FIG. 14 is a block diagram illustrating an example of a detailedconfiguration of the inverse quantization section 63 of the imagedecoding device 60 illustrated in FIG. 12. Referring to FIG. 14, theinverse quantization section 63 includes a quantization matrix settingsection 232 and an inverse quantization computing section 234.

(1) Quantization Matrix Setting Section

The quantization matrix setting section 232 recognizes the shapes andsizes of the transform units used during inverse orthogonal transform bythe inverse orthogonal transform section 64, and sets a quantizationmatrix corresponding to the recognized shape and size for each transformunit. For example, the quantization matrix setting section 232 acquirestransform unit information included in the header information of anencoded stream. Then, the quantization matrix setting section 232recognizes the shape and size of each transform unit from the transformunit information, and sets, for each transform unit, a quantizationmatrix corresponding to the recognized shape and size from among thequantization matrices generated by the generation section 214 of thesyntax processing section 61. Note that the quantization matrix settingsection 232 may also set for each transform unit a quantization matrixthat differs for every combination of prediction mode (intraprediction/inter prediction) and signal component (Y/Cb/Cr), forexample.

(2) Inverse Quantization Somputing Section

The inverse quantization computing section 234 uses the quantizationmatrix set by the quantization matrix setting section 232 to inverselyquantize the transform coefficient data (quantized data) input from theorthogonal transform section 62 for each transform unit. The inversequantization computing section 234 then outputs inversely quantizedtransform coefficient data to the inverse orthogonal transform section64.

[3-4. Exemplary Configuration of Inverse Orthogonal Transform Section]

FIG. 15 is a block diagram illustrating an example of a detailedconfiguration of the inverse orthogonal transform section 64 of theimage decoding device 60 illustrated in FIG. 12. Referring to FIG. 15,the inverse orthogonal transform section 64 includes a transform unitsetting section 242 and an inverse orthogonal transform computingsection 244.

(1) Transform Unit Setting Section

The transform unit setting section 242 sets square or non-squaretransform units as the transform units used when inverse orthogonallytransforming image data to decode. The shape of a transform unit set bythe transform unit setting section 242 may be square or non-square. Forexample, in the case where the intra prediction section 80 uses theshort distance intra prediction method discussed earlier, the transformunit setting section 242 may set a non-square transform unit of the samesize as a prediction unit in an image when a non-square prediction unitis selected as the prediction unit.

(2) Orthogonal Transform Computing Section

The inverse orthogonal transform computing section 244 generatesprediction error data by inverse orthogonally transforming transformcoefficient data input from the inverse quantization section 63 for eachtransform unit set by the transform unit setting section 242. Theinverse orthogonal transform computing section 244 then outputs thegenerated prediction error data to the addition section 65.

4. Process Flow During Decoding According to Embodiment (1) Overview ofProcess Flow

FIG. 16 is a flowchart illustrating an example of the flow of a processduring decoding by an image decoding device 60 according to the presentembodiment. Note that for the sake of the clarity of the descriptionherein, only the processing steps relating to the inverse quantizationand inverse orthogonal transform of image data are illustrated.

Referring to FIG. 16, first, the parameter acquisition section 212 ofthe syntax processing section 61 recognizes header information such asSPSs, PPSs, and slice headers from the stream of image data, andacquires quantization matrix parameters included in the headerinformation (step S210).

Next, the generation section 214 of the syntax processing section 61generates a square quantization matrix from among quantization matrixcandidates which may be used by the inverse quantization section 63, onthe basis of the quantization matrix parameters acquired by theparameter acquisition section 212 (step S220).

Next, the generation section 214 generates a non-square quantizationmatrix from among quantization matrix candidates which may be used bythe inverse quantization section 63, on the basis of the abovequantization matrix parameters (step S230). A detailed flow of theprocess at this point will be further described later using FIG. 17.

Next, the quantization matrix setting section 232 of the inversequantization section 63 sets, for each transform unit, a quantizationmatrix corresponding to the combination of the shape and size of thattransform unit (step S260).

Next, the inverse quantization computing section 234 of the inversequantization section 63 uses the quantization matrix set by thequantization matrix setting section 232 to inversely quantize thequantized data input from the lossless decoding section 62 for eachtransform unit (step S270).

Next, the inverse quantization computing section 234 of the inversequantization section 63 uses the quantization matrix set by thequantization matrix setting section 232 to inversely quantize thequantized data input from the lossless decoding section 62 for eachtransform unit (step S270).

Next, the inverse orthogonal transform section 64 generates predictionerror data by inverse orthogonally transforming transform coefficientdata input from the inverse quantization section 63 for each transformunit (step S280). By having the addition section 65 add the predictionerror data generated at this point to the predictive image data, theimage data from before encoding is restored.

Note that the processing from step S260 to step S280 typically may berepeated for all transform units within the image to decode.

(2) Non-Square Quantization Matrix Generation Process

FIG. 17 is a flowchart illustrating an example of the flow of anon-square quantization matrix generation process in step S230 of FIG.16. The process illustrated in FIG. 17 may be conducted for eachparameter set that includes quantization matrix parameters. Note thateach parameter set is assumed to include quantization matrix parameterslike those exemplified in FIG. 5.

Referring to FIG. 17, first, the generation section 214 acquires thenon-square matrix flag (step S231). The generation section 214 thendetermines whether or not to generate a non-square quantization matrix,on the basis of the value of the non-square matrix flag (step S232). Atthis point, the generation section 214 skips subsequent processing inthe case of determining to not generate a non-square quantizationmatrix. On the other hand, the process proceeds to step S234 in the casewhere the generation section 214 determines to generate a non-squarequantization matrix.

The processing from step S236 to step S252 may be repeated for everytype of non-square quantization matrix (step S234). Types of non-squarequantization matrices may be classified by combinations of quantizationmatrix size, prediction mode, and signal components, for example.

In step S238, the generation section 214 determines whether or not thecorresponding square quantization matrix is a default quantizationmatrix (step S236). At this point, the process proceeds to step S237 inthe case where the corresponding square quantization matrix is a defaultquantization matrix. On the other hand, the process proceeds to stepS238 in the case where the corresponding square quantization matrix isnot a default quantization matrix.

In step S237, the generation section 214 acquires a predefined, defaultquantization matrix being stored in memory as the non-squarequantization matrix to be generated (step S237).

In step S238, the generation section 214 acquires the generation mode(step S238). The generation section 214 then switches the subsequentprocessing on the basis of the value of the acquired generation mode.

For example, in the case where the generation mode indicates full scanmode (step S240), the generation section 214 additionally acquiresdifferential data, and generates a non-square quantization matrix infull scan mode (step S242).

Additionally, in the case where the generation mode indicates copy mode(step S244), the generation section 214 additionally acquires thedesignation mode, and if necessary, the copy source ID and the copysource direction (step S246). The generation section 214 then generatesa non-square quantization matrix in copy mode, or in other words, bycopying designated rows or columns from a corresponding squarequantization matrix (step S248).

Additionally, in the case where the generation mode indicates transposemode, for example, the generation section 214 generates a non-squarequantization matrix in transpose mode from another non-squarequantization matrix having a symmetric shape to the non-squarequantization matrix (step S250).

Note that in the case of predefined generation of a non-squarequantization matrix by copying from a square quantization matrix, anon-square quantization matrix may be generated in copy mode as ageneral rule, without switching processing on the basis of the value ofthe generation mode.

Furthermore, in the case where residual data exists in copy mode ortranspose mode, the generation section 214 adds residual to each elementof the copied quantization matrix or the transposed quantization matrix(step S252).

5. Application to Various Codecs

Technology according to the present disclosure is applicable to variouscodecs related to image encoding and decoding. In this section, exampleswill be described in which technology according to the presentdisclosure is respectively applied to a multiview codec and a scalablecodec.

[5-1. Multiview Codec]

A multiview codec is an image coding scheme for encoding and decodingwhat is called multiview video. FIG. 18 is an explanatory diagram forexplaining a multiview codec. Referring to FIG. 18, frame sequences forthree views respectively captured from three viewpoints are illustrated.Each view is given a view ID (view_id). One of these multiple views isdesignated the base view. The views other than the base view are callednon-base views. In the example in FIG. 18, the view with the view ID “0”is the base view, while the two views with the view IDs “1” and “2” arenon-base views. When encoding this multiview image data, the overalldata size of the encoded stream may be compressed by encoding frames inthe non-base views on the basis of encoding information for frames inthe base view.

In an encoding process and decoding process according to the abovemultiview codec, quantization matrices corresponding to non-squaretransform units may be generated from quantization matricescorresponding to square transform units. Control parameters of some kind(the parameters exemplified in FIG. 5, for example) may also be set foreach view in order to generate the quantization matrices to be used witheach view. Also, control parameters set in the base view may be reusedfor non-base views. Also, a flag indicating whether or not to reusecontrol parameters across views may be additionally specified.

FIG. 19 is an explanatory diagram for explaining an application of theearlier image encoding process to a multiview codec. Referring to FIG.19, a configuration of a multiview encoding device 710 is illustrated asan example. The multiview encoding device 710 is equipped with a firstencoding section 720, a second encoding section 730, and a multiplexingsection 740.

The first encoding section 720 encodes base view images, and generatesan encoded stream of the base view. The second encoding section 730encodes non-base view images, and generates encoded streams of thenon-base views. The multiplexing section 740 multiplexes the encodedstream of the base view generated by the first encoding section 720 withone or more encoded streams of non-base views generated by the secondencoding section 730, and generates a multiview multiplexed stream.

The first encoding section 720 and the second encoding section 730exemplified in FIG. 19 have a similar configuration to the imageencoding device 10 according to an embodiment discussed earlier. Thus,it becomes possible to generate quantization matrices corresponding tonon-square transform units to be used for each view from quantizationmatrices corresponding to square transform units. Parameters thatcontrol these processes may be inserted into the header area of theencoded stream for each view, or inserted into a shared header area inthe multiplexed stream.

FIG. 20 is an explanatory diagram for explaining an application of theearlier image decoding process to a multiview codec. Referring to FIG.20, a configuration of a multiview decoding device 760 is illustrated asan example. The multiview decoding device 760 is equipped with ademultiplexing section 770, a first decoding section 780, and a seconddecoding section 790.

The demultiplexing section 770 demultiplexes a multiview multiplexedstream into an encoded stream of a base view and one or more encodedstreams of non-base views. The first decoding section 780 decodes theencoded stream of the base view into base view images. The seconddecoding section 790 decodes the encoded streams of the non-base viewsinto non-base view images.

The first decoding section 780 and the second decoding section 790exemplified in FIG. 20 have a similar configuration to the imagedecoding device 60 according to an embodiment discussed earlier. Thus,it becomes possible to generate quantization matrices corresponding tonon-square transform units to be used for each view from quantizationmatrices corresponding to square transform units. Parameters thatcontrol these processes may be acquired from the header area of theencoded stream for each view, or acquired from a shared header area inthe multiplexed stream.

[5-2. Scalable Codec]

A scalable codec is an image coding scheme for realizing what is calledscalable coding. FIG. 21 is an explanatory diagram for explaining ascalable codec. Referring to FIG. 21, frame sequences for three layersthat differ in spatial resolution, temporal resolution, or quality areillustrated. Each layer is given a layer ID (layer_id). Of thesemultiple layers, the layer with the lowest resolution (or quality) isthe base layer. The layers other than the base layer are calledenhancement layers. In the example in FIG. 21, the layer with the layerID “0” is the base layer, while the two layers with the layer IDs “1”and “2” are enhancement layers. When encoding this multilayer imagedata, the overall data size of the encoded stream may be compressed byencoding frames in the enhancement layers on the basis of encodinginformation for frames in the base layer.

In an encoding process and decoding process according to the abovescalable codec, quantization matrices corresponding to non-squaretransform units may be generated from quantization matricescorresponding to square transform units. Control parameters of some kind(the parameters exemplified in FIG. 5, for example) may also be set foreach layer in order to generate the quantization matrices to be usedwith each layer. Also, control parameters set in the base layer may bereused for enhancement layers. Also, a flag indicating whether or not toreuse control parameters across layers may be additionally specified.

FIG. 22 is an explanatory diagram for explaining an application of theearlier image encoding process to a scalable codec. Referring to FIG.22, a configuration of a scalable encoding device 810 is illustrated asan example. The scalable encoding device 810 is equipped with a firstencoding section 820, a second encoding section 830, and a multiplexingsection 840.

The first encoding section 820 encodes base layer images, and generatesan encoded stream of the base layer. The second encoding section 830encodes enhancement layer images, and generates encoded streams of theenhancement layers. The multiplexing section 840 multiplexes the encodedstream of the base layer generated by the first encoding section 820with the encoded streams of one or more enhancement layers generated bythe second encoding section 830, and generates a multilayer multiplexedstream.

The first encoding section 820 and the second encoding section 830exemplified in FIG. 22 have a similar configuration to the imageencoding device 10 according to an embodiment discussed earlier. Thus,it becomes possible to generate quantization matrices corresponding tonon-square transform units to be used for each layer from quantizationmatrices corresponding to square transform units. Parameters thatcontrol these processes may be inserted into the header area of theencoded stream for each layer, or inserted into a shared header area inthe multiplexed stream.

FIG. 23 is an explanatory diagram for explaining an application of theearlier image decoding process to a scalable codec. Referring to FIG.23, a configuration of a scalable decoding device 860 is illustrated asan example. The scalable decoding device 860 is equipped with ademultiplexing section 870, a first decoding section 880, and a seconddecoding section 890.

The demultiplexing section 870 demultiplexes a multilayer multiplexedstream into an encoded stream of a base layer and one or more encodedstreams of enhancement layers. The first decoding section 880 decodesthe encoded stream of the base layer into base layer images. The seconddecoding section 890 decodes the encoded streams of the enhancementlayers into enhancement layer images.

The first decoding section 880 and the second decoding section 890exemplified in FIG. 23 have a similar configuration to the imagedecoding device 60 according to an embodiment discussed earlier. Thus,it becomes possible to generate quantization matrices corresponding tonon-square transform units to be used for each layer from quantizationmatrices corresponding to square transform units. Parameters thatcontrol these processes may be acquired from the header area of theencoded stream for each layer, or acquired from a shared header area inthe multiplexed stream.

6. Example Application

The image encoding device 10 and the image decoding device 60 accordingto the embodiment described above may be applied to various electronicappliances such as a transmitter and a receiver for satellitebroadcasting, cable broadcasting such as cable TV, distribution on theInternet, distribution to client devices via cellular communication, andthe like, a recording device that records images onto a medium such asan optical disc, a magnetic disk, or flash memory, and a playback devicethat plays back images from such storage media. Four exampleapplications will be described below.

[6-1. First Example Application]

FIG. 24 is a block diagram illustrating an exemplary schematicconfiguration of a television adopting the embodiment described above. Atelevision 900 includes an antenna 901, a tuner 902, a demultiplexer903, a decoder 904, a video signal processing section 905, a displaysection 906, an audio signal processing section 907, a speaker 908, anexternal interface 909, a control section 910, a user interface 911, anda bus 912.

The tuner 902 extracts a signal of a desired channel from broadcastsignals received via the antenna 901, and demodulates the extractedsignal. Then, the tuner 902 outputs an encoded bit stream obtained bydemodulation to the demultiplexer 903. That is, the tuner 902 serves astransmission means of the television 900 for receiving an encoded streamin which an image is encoded.

The demultiplexer 903 separates a video stream and an audio stream of aprogram to be viewed from the encoded bit stream, and outputs theseparated streams to the decoder 904. Also, the demultiplexer 903extracts auxiliary data such as an electronic program guide (EPG) fromthe encoded bit stream, and supplies the extracted data to the controlsection 910. Additionally, the demultiplexer 903 may performdescrambling in the case where the encoded bit stream is scrambled.

The decoder 904 decodes the video stream and the audio stream input fromthe demultiplexer 903. Then, the decoder 904 outputs video datagenerated by the decoding process to the video signal processing section905. Also, the decoder 904 outputs the audio data generated by thedecoding process to the audio signal processing section 907.

The video signal processing section 905 plays back the video data inputfrom the decoder 904, and causes the display section 906 to display thevideo. The video signal processing section 905 may also cause thedisplay section 906 to display an application screen supplied via anetwork. Further, the video signal processing section 905 may performadditional processes such as noise removal, for example, on the videodata according to settings. Furthermore, the video signal processingsection 905 may generate graphical user interface (GUI) images such asmenus, buttons, or a cursor, for example, and superimpose the generatedimages onto an output image.

The display section 906 is driven by a drive signal supplied by thevideo signal processing section 905, and displays a video or an image ona video screen of a display device (such as a liquid crystal display, aplasma display, or an OLED display, for example).

The audio signal processing section 907 performs playback processes suchas D/A conversion and amplification on the audio data input from thedecoder 904, and outputs audio from the speaker 908. Also, the audiosignal processing section 907 may perform additional processes such asnoise removal on the audio data.

The external interface 909 is an interface for connecting the television900 to an external appliance or a network. For example, a video streamor an audio stream received via the external interface 909 may bedecoded by the decoder 904. That is, the external interface 909 alsoserves as transmission means of the televisions 900 for receiving anencoded stream in which an image is encoded.

The control section 910 includes a processor such as a centralprocessing unit (CPU), and memory such as random access memory (RAM),and read-only memory (ROM). The memory stores a program to be executedby the CPU, program data, EPG data, data acquired via a network, and thelike. The program stored in the memory is read and executed by the CPUwhen activating the television 900, for example. By executing theprogram, the CPU controls the operation of the television 900 accordingto an operation signal input from the user interface 911, for example.

The user interface 911 is connected to the control section 910. The userinterface 911 includes buttons and switches used by a user to operatethe television 900, and a remote control signal receiver, for example.The user interface 911 detects an operation by the user via thesestructural elements, generates an operation signal, and outputs thegenerated operation signal to the control section 910.

The bus 912 interconnects the tuner 902, the demultiplexer 903, thedecoder 904, the video signal processing section 905, the audio signalprocessing section 907, the external interface 909, and the controlsection 910.

In a television 900 configured in this way, the decoder 904 includes thefunctions of an image decoding device 60 according to the foregoingembodiments. Consequently, it is possible to suppresses increases in thebit rate required to define quantization matrices for video to bedecoded by the television 900, even in cases where non-square transformunits may be used.

[6-2. Second Example Application]

FIG. 25 is a block diagram illustrating an exemplary schematicconfiguration of a mobile phone adopting the embodiment described above.A mobile phone 920 includes an antenna 921, a communication section 922,an audio codec 923, a speaker 924, a microphone 925, a camera section926, an image processing section 927, a multiplexing/demultiplexing(mux/demux) section 928, a recording and playback section 929, a displaysection 930, a control section 931, an operable section 932, and a bus933.

The antenna 921 is connected to the communication section 922. Thespeaker 924 and the microphone 925 are connected to the audio codec 923.The operable section 932 is connected to the control section 931. Thebus 933 interconnects the communication section 922, the audio codec923, the camera section 926, the image processing section 927, themux/demux section 928, the recording and playback section 929, thedisplay 930, and the control section 931.

The mobile phone 920 performs operations such as transmitting andreceiving audio signals, transmitting and receiving emails or imagedata, taking images, and recording data in various operating modesincluding an audio communication mode, a data communication mode, animaging mode, and a videophone mode.

In the audio communication mode, an analog audio signal generated by themicrophone 925 is supplied to the audio codec 923. The audio codec 923converts the analog audio signal into audio data, and A/D converts andcompresses the converted audio data. Then, the audio codec 923 outputsthe compressed audio data to the communication section 922. Thecommunication section 922 encodes and modulates the audio data, andgenerates a transmit signal. Then, the communication section 922transmits the generated transmit signal to a base station (notillustrated) via the antenna 921. Also, the communication section 922amplifies a wireless signal received via the antenna 921 and convertsthe frequency of the wireless signal, and acquires a received signal.Then, the communication section 922 demodulates and decodes the receivedsignal and generates audio data, and outputs the generated audio data tothe audio codec 923. The audio codec 923 decompresses and D/A convertsthe audio data, and generates an analog audio signal. Then, the audiocodec 923 supplies the generated audio signal to the speaker 924 andcauses audio to be output.

Also, in the data communication mode, the control section 931 generatestext data that makes up an email, according to operations by a user viathe operable section 932, for example. Moreover, the control section 931causes the text to be displayed on the display section 930. Furthermore,the control section 931 generates email data according to transmitinstructions from the user via the operable section 932, and outputs thegenerated email data to the communication section 922. The communicationsection 922 encodes and modulates the email data, and generates atransmit signal. Then, the communication section 922 transmits thegenerated transmit signal to a base station (not illustrated) via theantenna 921. Also, the communication section 922 amplifies a wirelesssignal received via the antenna 921 and converts the frequency of thewireless signal, and acquires a received signal. Then, the communicationsection 922 demodulates and decodes the received signal, reconstructsthe email data, and outputs the reconstructed email data to the controlsection 931. The control section 931 causes the display section 930 todisplay the contents of the email, and also causes the email data to bestored in the storage medium of the recording and playback section 929.

The recording and playback section 929 includes an arbitrary readableand writable storage medium. For example, the storage medium may be abuilt-in storage medium such as RAM, or flash memory, or an externallymounted storage medium such as a hard disk, a magnetic disk, amagneto-optical disc, an optical disc, USB memory, or a memory card.

Furthermore, in the imaging mode, the camera section 926 takes an imageof a subject, generates image data, and outputs the generated image datato the image processing section 927, for example. The image processingsection 927 encodes the image data input from the camera section 926,and causes the encoded stream to be stored in the storage medium of therecording and playback section 929.

Furthermore, in the videophone mode, the mux/demux section 928multiplexes a video stream encoded by the image processing section 927and an audio stream input from the audio codec 923, and outputs themultiplexed stream to the communication section 922, for example. Thecommunication section 922 encodes and modulates the stream, andgenerates a transmit signal. Then, the communication section 922transmits the generated transmit signal to a base station (notillustrated) via the antenna 921. Also, the communication section 922amplifies a wireless signal received via the antenna 921 and convertsthe frequency of the wireless signal, and acquires a received signal.The transmit signal and received signal may include an encoded bitstream. Then, the communication section 922 demodulates and decodes thereceived signal, reconstructs the stream, and outputs the reconstructedstream to the mux/demux section 928. The mux/demux section 928 separatesa video stream and an audio stream from the input stream, and outputsthe video stream to the image processing section 927 and the audiostream to the audio codec 923. The image processing section 927 decodesthe video stream, and generates video data. The video data is suppliedto the display section 930, and a series of images is displayed by thedisplay section 930. The audio codec 923 decompresses and D/A convertsthe audio stream, and generates an analog audio signal. Then, the audiocodec 923 supplies the generated audio signal to the speaker 924 andcauses audio to be output.

In a mobile phone 920 configured in this way, the image processingsection 927 includes the functions of the image encoding device 10 andthe image decoding device 60 according to the foregoing embodiments.Consequently, it is possible to suppresses increases in the bit raterequired to define quantization matrices for video encoded and decodedby the mobile phone 920, even in cases where non-square transform unitsmay be used.

[6-3. Third Example Application]

FIG. 26 is a block diagram illustrating an exemplary schematicconfiguration of a recording and playback device adopting the embodimentdescribed above. A recording and playback device 940 encodes, andrecords onto a recording medium, the audio data and video data of areceived broadcast program, for example. The recording and playbackdevice 940 may also encode, and record onto the recording medium, audiodata and video data acquired from another device, for example.Furthermore, the recording and playback device 940 plays back datarecorded onto the recording medium via a monitor and speaker accordingto instructions from a user, for example. At such times, the recordingand playback device 940 decodes the audio data and the video data.

The recording and playback device 940 includes a tuner 941, an externalinterface 942, an encoder 943, a hard disk drive (HDD) 944, a disc drive945, a selector 946, a decoder 947, an on-screen display (OSD) 948, acontrol section 949, and a user interface 950.

The tuner 941 extracts a signal of a desired channel from broadcastsignals received via an antenna (not illustrated), and demodulates theextracted signal. Then, the tuner 941 outputs an encoded bit streamobtained by demodulation to the selector 946. That is, the tuner 941serves as transmission means of the recording and playback device 940.

The external interface 942 is an interface for connecting the recordingand playback device 940 to an external appliance or a network. Forexample, the external interface 942 may be an IEEE 1394 interface, anetwork interface, a USB interface, a flash memory interface, or thelike. For example, video data and audio data received by the externalinterface 942 are input into the encoder 943. That is, the externalinterface 942 serves as transmission means of the recording and playbackdevice 940.

In the case where the video data and the audio data input from theexternal interface 942 are not encoded, the encoder 943 encodes thevideo data and the audio data. Then, the encoder 943 outputs the encodedbit stream to the selector 946.

The HDD 944 records onto an internal hard disk an encoded bit stream,which is compressed content data such as video or audio, variousprograms, and other data. Also, the HDD 944 reads such data from thehard disk when playing back video and audio.

The disc drive 945 records or reads data with respect to an insertedrecording medium. The recording medium inserted into the disc drive 945may be a DVD disc (such as a DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD+, orDVD+RW disc), a Blu-ray (registered trademark) disc, or the like, forexample.

When recording video and audio, the selector 946 selects an encoded bitstream input from the tuner 941 or the encoder 943, and outputs theselected encoded bit stream to the HDD 944 or the disc drive 945. Also,when playing back video and audio, the selector 946 outputs an encodedbit stream input from the HDD 944 or the disc drive 945 to the decoder947.

The decoder 947 decodes the encoded bit stream, and generates video dataand audio data. Then, the decoder 947 outputs the generated video datato the OSD 948. Also, the decoder 904 outputs the generated audio datato an external speaker.

The OSD 948 plays back the video data input from the decoder 947, anddisplays video. Also, the OSD 948 may superimpose GUI images, such asmenus, buttons, or a cursor, for example, onto displayed video.

The control section 949 includes a processor such as a CPU, and memorysuch as RAM or ROM. The memory stores a program to be executed by theCPU, program data, and the like. A program stored in the memory is readand executed by the CPU when activating the recording and playbackdevice 940, for example. By executing the program, the CPU controls theoperation of the recording and playback device 940 according to anoperation signal input from the user interface 950, for example.

The user interface 950 is connected to the control section 949. The userinterface 950 includes buttons and switches used by a user to operatethe recording and playback device 940, and a remote control signalreceiver, for example. The user interface 950 detects an operation bythe user via these structural elements, generates an operation signal,and outputs the generated operation signal to the control section 949.

In a recording and playback device 940 configured in this way, theencoder 943 includes the functions of the image encoding device 10according to the foregoing embodiments. In addition, the decoder 947includes the functions of the image decoding device 60 according to theforegoing embodiments. Consequently, it is possible to suppressesincreases in the bit rate required to define quantization matrices forvideo encoded and decoded by the recording and playback device 940, evenin cases where non-square transform units may be used.

[6-4. Fourth Example Application]

FIG. 27 is a block diagram showing an example of a schematicconfiguration of an imaging device adopting the embodiment describedabove. An imaging device 960 takes an image of a subject, generates animage, encodes the image data, and records the image data onto arecording medium.

The imaging device 960 includes an optical block 961, an imaging section962, a signal processing section 963, an image processing section 964, adisplay section 965, an external interface 966, memory 967, a mediadrive 968, an OSD 969, a control section 970, a user interface 971, anda bus 972.

The optical block 961 is connected to the imaging section 962. Theimaging section 962 is connected to the signal processing section 963.The display section 965 is connected to the image processing section964. The user interface 971 is connected to the control section 970. Thebus 972 interconnects the image processing section 964, the externalinterface 966, the memory 967, the media drive 968, the OSD 969, and thecontrol section 970.

The optical block 961 includes a focus lens, an aperture stop mechanism,and the like. The optical block 961 forms an optical image of a subjecton the imaging surface of the imaging section 962. The imaging section962 includes an image sensor such as a CCD or CMOS sensor, andphotoelectrically converts the optical image formed on the imagingsurface into an image signal which is an electrical signal. Then, theimaging section 962 outputs the image signal to the signal processingsection 963.

The signal processing section 963 performs various camera signalprocesses such as knee correction, gamma correction, and colorcorrection on the image signal input from the imaging section 962. Thesignal processing section 963 outputs the processed image data to theimage processing section 964.

The image processing section 964 encodes the image data input from thesignal processing section 963, and generates encoded data. Then, theimage processing section 964 outputs the encoded data thus generated tothe external interface 966 or the media drive 968. Also, the imageprocessing section 964 decodes encoded data input from the externalinterface 966 or the media drive 968, and generates image data. Then,the image processing section 964 outputs the generated image data to thedisplay section 965. Also, the image processing section 964 may outputthe image data input from the signal processing section 963 to thedisplay section 965, and cause the image to be displayed. Furthermore,the image processing section 964 may superimpose display data acquiredfrom the OSD 969 onto an image to be output to the display section 965.

The OSD 969 generates GUI images such as menus, buttons, or a cursor,for example, and outputs the generated images to the image processingsection 964.

The external interface 966 is configured as an USB input/outputterminal, for example. The external interface 966 connects the imagingdevice 960 to a printer when printing an image, for example. Also, adrive is connected to the external interface 966 as necessary. Aremovable medium such as a magnetic disk or an optical disc, forexample, is inserted into the drive, and a program read from theremovable medium may be installed in the imaging device 960.Furthermore, the external interface 966 may be configured as a networkinterface to be connected to a network such as a LAN or the Internet.That is, the external interface 966 serves as transmission means of theimage capturing device 960.

A recording medium to be inserted into the media drive 968 may be anarbitrary readable and writable removable medium, such as a magneticdisk, a magneto-optical disc, an optical disc, or semiconductor memory,for example. Also, a recording medium may be permanently installed inthe media drive 968 to constitute a non-portable storage section such asan internal hard disk drive or a solid-state drive (SSD), for example.

The control section 970 includes a processor such as a CPU, and memorysuch as RAM or ROM. The memory stores a program to be executed by theCPU, program data, and the like. A program stored in the memory is readand executed by the CPU when activating the imaging device 960, forexample. By executing the program, the CPU controls the operation of theimaging device 960 according to an operation signal input from the userinterface 971, for example.

The user interface 971 is connected to the control section 970. The userinterface 971 includes buttons, switches and the like used by a user tooperate the imaging device 960, for example. The user interface 971detects an operation by the user via these structural elements,generates an operation signal, and outputs the generated operationsignal to the control section 970.

In an imaging device 960 configured in this way, the image processingsection 964 includes the functions of the image encoding device 10 andthe image decoding device 60 according to the foregoing embodiments.Consequently, it is possible to suppresses increases in the bit raterequired to define quantization matrices for video encoded and decodedby the imaging device 960, even in cases where non-square transformunits may be used.

7. Conclusion

The foregoing uses FIGS. 1, 2, 3, 4, 5, 6A, 6B, 6C, 7, 8A, 8B, 9, 10,11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26 and 27 todescribe an image encoding device 10 and an image decoding device 60according to an embodiment. According to the foregoing embodiments, whenquantizing and inversely quantizing transform coefficient data, aquantization matrix is set for each transform unit, in correspondencewith the shape of the transform unit. The shapes of transform units aresquare or non-square, and quantization matrices corresponding tonon-square transform units may be generated from quantization matricescorresponding to square transform units. Consequently, it is possible topartially omit the definition of quantization matrices corresponding tonon-square transform units, or efficiently (that it, with a lower bitrate) define quantization matrices corresponding to non-square transformunits. For this reason, it is possible to avoid a significant loweringof coding efficiency due to an increase in the selectable types oftransform units, even in the case of implementing a scheme enabling theselection of non-square transform units.

Also, according to the embodiments, a non-square quantization matrixwhose long edge is equal in size to an edge of a given squarequantization matrix may be generated by copying some rows or columnsfrom the square quantization matrix. Consequently, it is possible toeasily generate a non-square quantization matrix by only repeating theextremely low-cost processing operation of copying element values.

Also, according to the embodiments, the rows or columns to copy from asquare quantization matrix may be flexibly specified in a parameter setor header of the encoded stream. Consequently, it is possible, via theabove copying, to generate quantization matrices suitable for quantizingand inversely quantizing the transform coefficient data of non-squaretransform units. Meanwhile, in the case where the rows and columns tocopy from a square quantization matrix are predefined, it is possible todecrease the number of quantization matrix parameters to encode andreduce transmission overhead, while also lowering device complexity.

Note that this specification describes an example in which thequantization matrix parameter multiplexed into the header of the encodedstream and transmitted from the encoding side to the decoding side.However, the technique of transmitting the quantization matrix parameteris not limited to such an example. For example, header information mayalso be transmitted or recorded as separate data associated with anencoded bit stream without being multiplexed into the encoded bitstream. Herein, the term “associated” means that images included in thebit stream (also encompassing partial images such as slices or blocks)and information corresponding to those images can be linked at the timeof decoding. In other words, information may also be transmitted on aseparate transmission channel from an image (or bit stream). Also, theinformation may be recorded to a separate recording medium (or aseparate recording area on the same recording medium) from the image (orbit stream). Furthermore, information and images (or bit streams) may beassociated with each other in arbitrary units such as multiple frames,single frames, or portions within frames, for example.

The foregoing thus describes preferred embodiments of the presentdisclosure in detail and with reference to the attached drawings.However, the technical scope of the present disclosure is not limited tosuch examples. It is clear to persons ordinarily skilled in thetechnical field to which the present disclosure belongs that variousmodifications or alterations may occur insofar as they are within thescope of the technical ideas stated in the claims, and it is to beunderstood that such modifications or alterations obviously belong tothe technical scope of the present disclosure. Additionally, the presenttechnology may also be configured as below.

(1)

An image processing device including: a decoding section that decodes anencoded stream and generates quantized transform coefficient data; and

an inverse quantization section that, taking transform coefficient dataas transform units to be used during inverse orthogonal transform,inversely quantizes the quantized transform coefficient data decoded bythe decoding section, such that in a case where a non-square transformunit is selected, the inverse quantization section uses a non-squarequantization matrix, corresponding to a non-square transform unit, thatis generated from a square quantization matrix corresponding to a squaretransform unit.

(2)

The image processing device according to (1), wherein

the non-square quantization matrix is generated by copying one of rowelements and column elements of the square quantization matrix.

(3)

The image processing device according to (2), wherein

a size of a long edge of the non-square quantization matrix is equal toa size of an edge of the square quantization matrix.

(4)

The image processing device according to (2) or (3), wherein

one of row elements and column elements to be copied from the squarequantization matrix are predefined.

(5)

The image processing device according to any one of (2) to (4), wherein

the non-square quantization matrix is generated by copying one of rowelements and column elements of the square quantization matrix at anequal interval.

(6)

The image processing device according to (5), wherein

an interval of one of row elements and column elements to be copied fromthe square quantization matrix is determined according to a ratio of asize of a short edge of the non-square quantization matrix versus a sizeof an edge of the square quantization matrix.

(7)

The image processing device according to (6), wherein

the ratio is 1:4, and the interval is one of four rows and four columns.

(8)

The image processing device according to (7), wherein

a size of the square quantization matrix is 4×4, and a size of thenon-square quantization matrix is one of 1×4 and 4×1.

(9)

The image processing device according to (7), wherein

a size of the square quantization matrix is 8×8, and a size of thenon-square quantization matrix is one of 2×8 and 8×2.

(10)

The image processing device according to (7), wherein

a size of the square quantization matrix is 16×16, and a size of thenon-square quantization matrix is one of 4×16 and 16×4.

(11)

The image processing device according to (7), wherein

a size of the square quantization matrix is 32×32, and a size of thenon-square quantization matrix is one of 8×32 and 32×8.

(12)

The image processing device according to any one of (2) to (11), furtherincluding:

a generation section that generates the non-square quantization matrixfrom the square quantization matrix.

(13)

The image processing device according to any one of (2) to (12), furtherincluding:

an inverse orthogonal transform section that inverse orthogonallytransforms the transform coefficient data inversely quantized by theinverse quantization section by using the selected non-square transformunit.

(14)

An image processing method including:

decoding an encoded stream and generating quantized transformcoefficient data; and

taking transform coefficient data as transform units to be used duringinverse orthogonal transform, inversely quantizing the decoded quantizedtransform coefficient data, such that in a case where a non-squaretransform unit is selected, a non-square quantization matrixcorresponding to a non-square transform unit is used, the non-squarequantization matrix being generated from a square quantization matrixcorresponding to a square transform unit.

REFERENCE SIGNS LIST

-   10 Image processing device (image encoding device)-   15 Orthogonal transform section-   152 Transform unit setting section-   16 Quantization section-   17 Lossless encoding section-   60 Image decoding device-   63 Inverse quantization section-   214 Generation section-   242 Transform unit setting section

What is claimed is:
 1. An image processing device, comprising: adecoding section configured to: decode an encoded stream; and generatequantized transform coefficient data based on the decoded stream; aninverse quantization section configured to: inversely quantize thequantized transform coefficient data corresponding to a non-squaretransform unit, wherein the quantized transform coefficient data isinversely quantized based on a non-square quantization matrix that isset by copy of one of row elements of a square quantization matrix orcolumn elements of the square quantization matrix, and wherein thesquare quantization matrix corresponds to a square transform unit; andgenerate first transform coefficient data based on the inversequantization of the quantized transform coefficient data; and an inversetransform section configured to inversely transform the first transformcoefficient data corresponding to the non-square transform unit.
 2. Theimage processing device according to claim 1, wherein a size of a longedge of the non-square quantization matrix is equal to a size of an edgeof the square quantization matrix.
 3. The image processing deviceaccording to claim 1, wherein the one of the row elements or the columnelements of the square quantization matrix correspond to a specificposition associated with the square quantization matrix.
 4. The imageprocessing device according to claim 1, wherein the non-squarequantization matrix is set based on the copy of the one of the rowelements or the column elements of the square quantization matrix at anequal interval.
 5. The image processing device according to claim 4,wherein an interval of the one of the row elements or the columnelements of the square quantization matrix is based on a ratio of a sizeof a short edge of the non-square quantization matrix and a size of anedge of the square quantization matrix.
 6. The image processing deviceaccording to claim 5, wherein the ratio is 1:4, and wherein the intervalis one of four rows or four columns.
 7. The image processing deviceaccording to claim 6, wherein a size of the square quantization matrixis 4×4, and a size of the non-square quantization matrix is one of 1×4or 4×1.
 8. The image processing device according to claim 6, wherein asize of the square quantization matrix is 8×8, and a size of thenon-square quantization matrix is one of 2×8 or 8×2.
 9. The imageprocessing device according to claim 6, wherein a size of the squarequantization matrix is 16×16, and a size of the non-square quantizationmatrix is one of 4×16 or 16×4.
 10. The image processing device accordingto claim 6, wherein a size of the square quantization matrix is 32×32,and a size of the non-square quantization matrix is one of 8×32 or 32×8.11. An image processing method, comprising: decoding an encoded stream;generating quantized transform coefficient data based on the decodedstream; inversely quantizing the quantized transform coefficient datacorresponding to a non-square transform unit, wherein the quantizedtransform coefficient data is inversely quantized based on a non-squarequantization matrix that is set based on a copy of one of row elementsof a square quantization matrix or column elements of the squarequantization matrix, wherein the square quantization matrix correspondsto a square transform unit; generating first transform coefficient databased on the inverse quantization of the quantized transform coefficientdata; and inversely transforming the first transform coefficient datacorresponding to the non-square transform unit.
 12. The image processingmethod according to claim 11, wherein a size of a long edge of thenon-square quantization matrix is equal to a size of an edge of thesquare quantization matrix.
 13. The image processing method according toclaim 11, wherein the one of the row elements or the column elements ofthe square quantization matrix correspond to a specific positionassociated with the square quantization matrix.
 14. The image processingmethod according to claim 11, wherein the non-square quantization matrixis set based on the copy of the one of the row elements or the columnelements of the square quantization matrix at an equal interval.
 15. Theimage processing method according to claim 14, wherein an interval ofthe one of the row elements or the column elements of the squarequantization matrix is based on a ratio of a size of a short edge of thenon-square quantization matrix and a size of an edge of the squarequantization matrix.
 16. The image processing method according to claim15, wherein the ratio is 1:4, and wherein the interval is one of fourrows or four columns.
 17. The image processing method according to claim16, wherein a size of the square quantization matrix is 4×4, and a sizeof the non-square quantization matrix is one of 1×4 or 4×1.
 18. Theimage processing method according to claim 16, wherein a size of thesquare quantization matrix is 8×8, and a size of the non-squarequantization matrix is one of 2×8 or 8×2.
 19. The image processingmethod according to claim 16, wherein a size of the square quantizationmatrix is 16×16, and a size of the non-square quantization matrix is oneof 4×16 or 16×4.
 20. The image processing method according to claim 16,wherein a size of the square quantization matrix is 32×32, and a size ofthe non-square quantization matrix is one of 8×32 or 32×8.